Method and apparatus for storage and retrieval of very large databases using a direct pipe

ABSTRACT

A method and apparatus for directly connecting very large data streams from an archive command into a backup data system using an “intelligent process.” An output stream is piped into an intelligent pipe-reading process and distributed over a set of temporary data stores built from raw storage resources. A pipe interface process supervises backup of each filled data store, while the remaining output stream continues to be piped into another available data store. The backup system completes archiving of the datastream, keeping a catalog of the datastream storage locations. To retrieve the data, the intelligent process is run in reverse as a pipe-writing process, requesting data from the backup system. Retrieved data traverses the data stores from the backup system and are pumped into the pipe-writing process for delivery to the pipe output file identified by the retrieve or import command.

FIELD OF THE INVENTION

This invention relates generally to storage and retrieval of largedatabases, and more particularly to the use of temporary storage formoving large files.

BACKGROUND OF THE INVENTION

As large computing enterprises continue their migration from centralizedmainframes and volumes of data on direct access storage devices (DASD)and serial tape drives, the reliance upon open systems databasetechnology has increased. The ability to quickly adapt mass storagesystems to new platforms, while retaining high performance andreliability will remain key elements for system integrators.

In earlier times, rooms full of DASD and tape drives were maintained bylegions of storage operators who physically mounted and unmounted tapesand disk packs and moved them to and from libraries according to dailyschedules and batch job instructions. Technology improvements allowedthe use of self-contained “mass storage” units, using robotic arms tomove archived storage media to and from the drive mechanisms in a matterof seconds. Further developments of storage media have enabled a cachemodel in which large masses of data are held in offline resources andsmaller portions can be uploaded to the high-speed cache as necessary.Data availability has also been increased through the use of arrays ofmirrored databases, either single or multi-threaded, for multiplesimultaneous access capabilities.

Even with mirrored systems, operational concerns often require that themagnetic or other storage media be archived or “backed up.” There arealso occasional system re-organizations or restructuring during which adatabase may be converted by copying it out in one format (“export”) andcopying it back in (“import”) with a different format, or into adifferent structure. Back-up issues will also arise when converting fromone database management system (DBMS) to another, or sharing databases.Application programs themselves may also request the operating system tomake a large “save” of data files, usually after a modification is madeto the file. Backing up of large volumes of data can be a time consumingand resource intensive operation. Mass storage systems,disadvantageously may be unavailable while large back-up operations areperformed.

FIG. 1 illustrates a typical system in which a host computer 10 isconnected to a backup system 12 and a storage system 14. Creating aphysical backup of the entire database 24 of the storage system 14 oftenrequires a large investment of time and resources. In an open, networkedarray of storage devices, a physical backup of a database may be handledby arranging for a DBMS 22, such as provided by Oracle Corporation, tocommunicate with a dedicated back-up system 12, such as the EMC DataManager, from EMC Corporation of Hopkinton, Mass. The DBMS system vendoroften supplies an Application Programming Interface (API) 20 that can beinstalled in the host computer to handle the scheduling of the regularbackups. The DBMS system typically reads the data from the database viathe local DASD interface (such as SCSI bus), and delivers a buffer ofdata through the API. The application running the backup may becustomized or optimized for the particular mass storage system selected,such as the EMC Data Manager (EDM) which is optimized to run with EMC'sSymmetrix storage system(s). The EMC backup application, or somethingsimilar, take the necessary steps to send the data over the networkusing a connection-oriented protocol such as Transmission ControlProtocol (TCP). The receiving backup system then sends the data to amass storage unit 18, such as to write the data to an archive tape 18A.The major drawbacks of physical backup include that logical structures,such as tables of data cannot be backed up. Further, data cannot betransferred between machines of differing operating systems.Additionally, data blocks cannot be reorganized into a more efficientlayout.

Many of the major DBMS companies also provide a more generalizedfacility in which the data is exported as a standardized file, such- asin ASCII format, as part of a so-called “logical backup.” The ASCIIformat permits the file to be imported into most other systems, withoutinsurmountable compatibility problems. However, presently DMBS companiesgenerally do not provide the API necessary for a customer to properlyhandle the data stream generated by the logical backup. The result isthat many of the DBMSs generate very large backup files that have to bestored locally until they can be written to an archive device.

To overcome this disadvantage, some customers create their own primitivesolution by attaching a physical tape drive to the machine. The logicalbackup data stream is then directed into a process that Unix calls a“pipe,” buffered, and then directed (“piped”) by another Unix commandsuch as one that writes the data to the local tape drive 16, a DASD, orother demountable, writable media. A Unix pipe can be thought of as aFIFO (first-in first-out) data file having one process writing data intoit serially and another process serially reading data out. When the pipeis “empty,” the reading process simply waits for more data to be writtenby the other process. Other non-Unix operating systems such as DOS andWindows NT emulate the Unix pipe in various ways with similar results.Logical data streams are thus directed from a database export intoanother process that disposes of the data to the physical storage media,thus freeing up storage resources.

This primitive solution has several disadvantages. For one thing, itrequires a physical tape drive 16 to be attached to the computer host 10generating the backup. Alternatively, the logical backup could be pipedto a command that writes the data onto disk or equivalent. However, thissolution would require each such machine to have huge amounts of excessstorage capacity. In either case, additional operations personnel mustbe assigned to handling the tapes and disks, and maintaining the drives.Extra storage devices, media libraries, and personnel also take up extraspace in the facility. Another alternative would be to pipe the logicalbackup data stream into the network interface and send it to a differentmachine having a DASD or tape. When dealing with very large databases,these solutions could break down entirely, due to the operationaldifficulties of maintaining the necessary physical media, or opennetwork connections.

The named pipe provides a standard mechanism that can be used byprocesses that do not have to share a common process origin forprocess-to-process-to-device transfers of large amounts of data. Thedata sent to the named pipe can be read by any 5 authorized process thatknows the name of the named pipe. In particular, named pipes are used inconjunction with the Oracle DBMS import/export utility to perform thelogical backup/restores necessary to restructure or reorganize verylarge data bases (VLDBs). Typically, the user creates a named pipe andruns an export utility specifying the named pipe as the output device.The DBMS sees the pipe as a regular file. Another process, including forexample Oracle DBMS commands such as dd (convert and copy a file), cpio(copy files in and out), rsh (execute command on a remote system), etc.,then reads from the other end of the pipe and writes the data to actualmedia or the network. This technique is used to write export data tolocal disk/tape or over the network to available disk/tape on anothermachine.

As mentioned, a disadvantage of the existing methods is the large amountof time it takes to perform backups, during which the database may bepartially or completely offline due to read/write interlocks. Some ofthis delay can be reduced by segmenting export/backup files, and runningseveral processes in parallel. Even though the logical backup processcan be segmented into parallel streams by some DBMSs, theimplementations may be proprietary and not necessarily adaptable forimport to another DBMS. Also, disadvantageously, a dedicated disk ordedicated tape is required. Further, in known implementations, there isan inability to catalog multiple versions.

VLDB reorganization and restructuring are major operations for whichthere is no known highly efficient solution. Current solutions do notuse the data management services of an Enterprise Data Management Tool(such as EDM). A VLDB administrator has the ability to divide largetables into separate physical partitions. The partitions can be backedup, stored, exported and imported separately from the rest of the table.Customers consider this a critical feature that should be heavily used.Existing backup systems and APIs do not have the ability to export orimport partitions in parallel using all available tape drives. Any APIsolution would be necessarily proprietary to the DBMS vendor, and notgeneralizable to other systems.

SUMMARY OF THE INVENTION

The present invention, provides a method and apparatus for directlyconnecting very large data streams from an archive command into a backupdata system using an “intelligent process.”

According to the invention, a pipe interface process supervises backupof each filled data store, while the remaining output stream continuesto be piped into another available data store. The backup systemcompletes archiving of the datastream, keeping a catalog of thedatastream storage locations. To retrieve the data, the intelligentprocess is run in reverse as a pipe-writing process, requesting datafrom the backup system. Retrieved data traverses the data stores fromthe backup system and are pumped into the pipe-writing process fordelivery to the pipe output file identified by the retrieve or importcommand.

An intelligent pipe process according to the present invention is aconfigurable client process that runs on each system that requires theenhanced backup or export services. The intelligent pipe is configuredto manage the distribution of the piped data stream to and frompre-selected data stores, and to trigger the backup system for furtherdata handling on output. The receiving process in the backup system isalso configured to wait for storage requests from the intelligent pipe,schedule the storage, update the catalog, and report completion statusback to the intelligent pipe.

In an alternative embodiment of the invention, many of the data-storemanagement tasks and backup scheduling can be handled by aself-contained backup system adapted for the purpose, leaving lessactivity for the intelligent pipe process, thus freeing up additionalresources in the host computer.

Features of the invention include a method and apparatus whereintemporary storage, communication, and archiving for output of a genericdatabase export command pipe is managed in a manner transparent to thecustomer operating the host computer. The customer is relieved of theneed to maintain additional large local storage devices for physical orlogical backups. The customer does not have to construct and maintainspecialized pipes or scripts for network backups, or for differentphysical tape devices or DASDs.

Implementing a pipe process rather than a specific export/importfunction reduces the number of proprietary restrictions andconfiguration difficulties that depend upon choice of databaseimplementation. The backup and archive methodology remains independentof the database structure or selected DBMS. Similarly, there is no needto work with a different API from each DBMS vendor, nor anycustomization of the Unix operating system.

A plurality parallel pipes according to the invention can be implementedfor faster transfers of segmented database backups, where that featureis implemented in the DBMS. Furthermore, the invention enables anyoutput-generating Unix-like process (application save, non-Oraclebackup/restore) to have its results sent to the existing backup system,thus freeing up local resources automatically. Similarly, the sameinvention can be operated in reverse, enabling any Unix-likeinput-receiving process to obtain its data directly from a mass storagebackup system. The temporary disk space needed to implement the datastores can also be designed as part of a backup system rather than thehost computer, freeing up computer resources.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other features and advantages of the present inventionwill be more fully understood from the following detailed description ofan illustrative embodiment, taken in conjunction with the accompanyingdrawing in which:

FIG. 1 is a block diagram for a conventional back-up system according tothe prior art.

FIG. 2 is a block diagram illustrating a system having enhancedfacilities according to the invention.

FIG. 3 is a flowchart showing additional details in the operation of adirect connect intelligent process according to the present invention.

FIG. 4 is a flowchart showing the operation of an import operationaccording to the invention.

DETAILED DESCRIPTION

The present invention provides a method and apparatus for directlyconnecting very large data streams from a DBMS into a backup system. Asillustrated in FIG. 2, an output data stream 38 from a database exportcommand of a DBMS 22 is piped into an intelligent pipe-reading process30 and distributed over a set of temporary data stores 34 built from rawstorage resources 36. When a data store reaches capacity, theintelligent process 30 signals the backup system 12 to begin a backup ofth at data store, while the remaining export data stream. is piped intoanother available temporary data store 34. Once the final data store hasreceived an end-of-pipe signal, the backup system begins completionprocessing of the export request (e.g., writing to tape device 18). Thebackup system keeps a logical backup catalog, and writes it to thebackup tape with the archived datastream. Once the archive tape activityis complete, the backup system signals back via the backup controlprocess 39 to the intelligent process 30. The data store is markedavailable by the intelligent process. This continues until the dbms 22closes the pipe 38. At this point the intelligent pipe process 30 willnotify the backup control process 39 that the final data should bebacked up. It will then wait until notified by the backup controlprocess 39 that all data stores 34, 36 are available (i.e., the backupsfor all stores are complete). The intelligent pipe process 30 signalsthe backup control process 39 that there is no more data to back up. Thebackup control process 39 commits all catalog data 37 tying theindividual data store backups together as a single logical backup.

To activate the intelligent process according to an embodiment of theinvention, a named pipe is created using the Unix command mknod. Thenext command issued is eb_dc_pipe if=named_pipe. This command executes aprocess that will wait until data is written to the pipe. The eb_dc_pipeis an intelligent pipe reading command. Then a database command isissued, such as the Oracle command exp file=named_pipe, with specifiedoptions. These commands can be encapsulated within a single command. Inthe event that the backup is initiated from the EDM graphical userinterface (GUI), these commands will be encapsulated into a singlecommand which will be placed onto the host and executed at backup time.This command starts the DBMS writing the data to the pipe and could alsobe any application program, including non-Oracle logical backups. Forexample, the Oracle export command causes the DBMS to write anASCII-encoded file, using normal Unix primitives. In this case, the DBMSoutput is sent to the named pipe which is read by an intelligentprocess, although it is read and written as a normal file.

During configuration of the eb_dc_pipe software, a set of data stores onthe host computer is identified and listed. The list is also provided tothe backup system for linkage between the client and the backup system.Appropriate steps must be taken to translate the storage references toconform to the view of the disk space from the backup system that may bedifferent from the client's view of the same disk space.

A sufficient size and number of data stores are preferably defined sothat export processing is never waiting for a data store to becomeavailable. Factors known to influence these calculations include thetransmission delay, average queuing delays for each component in theprocess, storage access time in the data stores and mass storage,logical and physical arrangement of the data storage, etc. Oneembodiment of the data stores could be implemented on a mirrored disk,although a non-mirrored approach is likely to have less overhead anddevice conflict.

A typical archive operation in accord with an embodiment of the presentinvention is illustrated in FIG. 3. The intelligent process is typicallyactivated when the pipe is created in anticipation of a database backupstream. When the archive operation is activated, the illustratedintelligent process reads the list of data stores 40 that have. beenallocated for this operation. Data is then read 42 from the outputstream generated by the command feeding the pipe (e.g., the DBMS exportfile command). The intelligent process checks 44 to see if there is anydata, and if there is data, a data store is selected 46. Several datastores can then be managed in a “round robin” scheme, whereby one isfilled, then the next is filled while the first is being emptied, and soforth. Each emptied data store is put back on the list of available datastores.

The selected data store is tested to see if it is already in use 48 orfull 50. If it is in use, then the next data store in the list isselected 46 and tested again. A data store that is not busy and not fullis written 52 with data from the pipe, and the next data is read 42 fromthe pipe. This process loops as long as there is data, selecting thesame data store, until the data store fills up. When the data store istested 50 as full, the intelligent process signals 54 the data managerin the backup system to retrieve and archive the data in the currentdata store. Since the data store is full, it will be marked 56 with a“backup pending” flag until such time as the data manager completes thestorage process and resets the flag.

One system that can be adapted to carry out many of the data storesfunctions is the EMC Data Manager (EDM) from EMC Corporation ofHopkinton, Mass. A description of this enterprise-wide client-serverbackup solution can be found in U.S. patent application Ser. No.09/052,208 filed Mar. 31, 1998, and incorporated herein by reference. Insome implementations it may not be necessary to manage the set of datastores as part of the intelligent process running in the host computer,since the backup system could do this transparently, making the transferappear more like a TCP transfer across a network. Additional EMCdocumentation can be found at the EMC web site or requested from themanufacturer regarding these commercially available products.

Operational connection between the data stores and the backup system canbe implemented according to known methods for moving data from a disk inone system to a disk in another, via direct channel such as a SmallComputer System Interface (SCSI) bus, or via a network, such as by usingTCP. By directly writing to Symmetrix API Socket as known in the art,the network can be bypassed with a logical backup. This configurationwould permit a customer to perform both a physical and a logical backupusing the same configuration.

In any event, a direct connect backup technology would also besufficient for the purpose of removing the data from the data stores andplacing it into an archive file. The data in each data store must alsobe tied to a single logical backup. Control information about the data,so-called “meta-data,” must also be conveyed to the backup system. In aconventional API, the vendor supplies useful cataloging information suchas the application name, and version, etc. In this case, since the pipeis merely a data structure into which a preceding process pumps data,there is little meta-data available. However, the process can beconfigured to recognize the name of the program sending it the data, thedate and time, the machine name, the socket identification, and otherinformation. Other programs can be created that will determine who issending the data.

Once the end of the data is read 44, i.e., the last DBMS output bufferis flushed, the named pipe signals 58 the data manager of the backupsystem to complete the archiving process. This may include operationssuch as writing the backup tapes with their appropriate catalogs. It isimportant for proper processing that the named pipe wait 60 for thebackup system to complete its operations, as signaled by the datamanager. Only then should the named pipe terminate with the properreturn code to the user (i.e., complete or failed). In some cases a Unixcommand process may take a branch depending upon pipe process returncodes when a process terminates.

To retrieve the data from an archive tape 18A, an operator executes theappropriate command to import a file, or restore a database. The inputto the command is generated by a named pipe running in the reverse orderfrom the export mode. The pipe direction could be set by an optionalcommand switch, for example, such that the eb_dc_pipe intelligentprocess is producing data from the archive rather than archiving data.In effect, data is then pumped out of the archive tapes in the backupsystem, through the data stores, and into the import file established bythe retrieve command.

According to the flow chart of FIG. 4, the retrieve operation of anillustrative embodiment of the present invention proceeds as follows.The intelligent process is typically activated when the pipe is createdin anticipation of a database retrieval stream. When the retrievaloperation is. activated, the intelligent process illustrated reads thelist of data stores 80 that have been allocated for this operation.

The backup system is notified 82 which logical backup to restore from.This is typically a date and time of a specific backup instance of thisconfiguration. The selection may be determined by browsing the backupcatalogs 37 with the backup system GUI, or by direct user specification.The intelligent pipe process will request that the backup system restorethe backup files into the data stores in the same sequence as they werebacked up in. There is no requirement that there be the same number ofdata stores at restore time that were used for the backup. As each datastore is filled, it will be written to the named pipe in sequence,presenting the DBMS application with a single logical stream of data.

The selected data store 84 is checked to see if it is in an uploadpending state 86. This means that a backup file has been completelyrestored into the data store, but the data has not yet been written tothe named pipe. If it is in upload pending state, it is then checked tosee if it contains the next backup in logical restore sequence 88. If itis, and there are no data stores currently being written to the pipe 90,then the contents of the data store are written to the pipe 92.Subsequently, the data store is now made available for new data 94. Theactual pipe writing process may go on asynchronously from therestoration of data into data stores. This will help ensure that thepipe writing process spends little time waiting for data to beavailable. If the data store is either not the next in logical restoresequence or there are any data stores already being written to the pipe,then another data store is selected.

Given a data store that is not in an upload pending state, it is thenchecked to see if it is currently being restored into 96. In this case,it cannot be touched until the backup system completes its restore 98.If, however, a check of the backup system reveals that the restore forthis data store has completed 98, then the restore pending designationis removed, and the data store is marked as being ready to be written tothe pipe 100.

If the data store was in neither upload pending 86 or restore pending 96status, the backup system is checked to see if there are any morerestores left to be scheduled for this logical restore 102. If morerestores need to be done 102, the backup system is notified to beginrestore of the next backup file into this data store 104. The data storeis marked as being the next one in the sequence 106.

This process continues iteratively until there are no more restores tobe done 102, and all data stores have been written in sequence to thepipe 108.

Although the present invention has been described according to anillustrative embodiment, it would be apparent to one of ordinary skillin the art that a more sophisticated implementation could makebeneficial use of a cache system in which the backup system cache wouldmatch the speed of the DBMS and the network, thus minimizing the delayattributable to the backup system internals. Additional optimization maybe achieved based upon additional details of a particularizedimplementation and configuration of the DBMS and the backup system, suchas the known buffer sizes, physical disk sizes, channel speeds, type andnumber of physical devices, formats for meta-data, etc.

As another alternative, it should be apparent that known buffermanagement methods other than the described “round robin” method can beapplied without changing the underlying invention. Furthermore, the datastores themselves may be implemented as pipe processes using logicalrather than particular physical storage facilities.

One skilled in the art should also recognize that the describedinvention has numerous applications in addition to streamlining theexport/import file process, since the invention provides a generalizedfacility for using named pipes to move large amounts of data between ahost computer and a backup system.

Furthermore, although particular divisions of functions are providedamong the various components identified herein, it should be appreciatedthat fimctions attributed to one device or process may be beneficiallyincorporated into a different or separate device or process. Similarly,the functional steps described herein may be modified with othersuitable algorithms or processes that accomplish functions similar tothose of the method and apparatus described without significantdeviation from the invention.

Although the invention has been described and shown with respect to anillustrative embodiment thereof, it should be appreciated that theforegoing and other various changes, omissions, and additions in theform and detail thereof could be implemented without changing theunderlying invention.

1. A method of directly connecting an output of a computer process to astorage device comprising the steps of: creating a FIFO pipe fed by saidoutput; emptying said FIFO pipe into an intelligent process; and storingdata from said intelligent process into said storage device; whereinsaid intelligent process includes at least one FIFO data store and eachof said at least one FIFO data stores conveys said data from said FIFOpipe to said storage device.
 2. The method of claim 1 in which said stepof storing data from said intelligent process into said storage devicefurther comprises the steps of: reading data from said FIFO pipe;selecting one of said at least one FIFO data stores to identify aselected data store; marking said selected data store as busy; writingsaid data read from said FIFO pipe into said selected data store; ifsaid selected data store is filled, and there is additional data in saidFIFO pipe, then selecting a next one of said at least one FIFO datastores, writing said additional data in said FIFO pipe into saidselected next one of said at least one FIFO data stores; and writingsaid data from said selected data store into said storage device; ifsaid selected data store is filled, and there is no additional data insaid FIFO pipe, then waiting until said FIFO pipe is closed; and writingsaid data from said selected data store into said storage device.
 3. Themethod of claim 2 in which said step of selecting one of said at leastone FIFO data store further comprises the steps of: reading a list ofavailable data stores; determining which of said available data storesis not busy; and identifying the determined available data store asbeing the selected data store.
 4. The method of claim 3 in which saidstep of determining which of said data stores is not busy furthercomprises the steps of: managing said list of available data stores as around robin list in which each data store identified as busy is omittedfrom the round robin list, and each data store that becomes not busy isadded to the end of the round robin list.
 5. The method of claim 1 inwhich said step of storing data from said intelligent process into saidstorage device further comprises the steps of: determining a sequence ofsaid at least one data store to be filled with data; filling saidsequence of said at least one data store; passing information regardingsaid sequence to said storage device; and conveying data from saidsequence of said at least one data store to said storage deviceaccording to said information.
 6. An apparatus for directly storingoutput from a computer process into a storage device comprising: saidcomputer process generating data to an output; a FIFO pipe fed by saidoutput; an intelligent process fed by said FIFO pipe; and said storagedevice for storing data from said intelligent process.
 7. The apparatusof claim 6 in which said intelligent process further includes: a set ofphysical storage locations arranged as a set of data stores; a selectorfor determining which of said data stores should be next filled; amonitor for determining when said data store is full; and a selector fordetermining which of said full data stores should be next fed to saidstorage device.
 8. The apparatus of claim 7 in which said selector isadapted to use a round robin list for selecting a next available datastore, and each data store is removed from said list while being filledor while being fed to said storage device.
 9. A method of directlyconnecting a computer processing output to a data storage devicecomprising the steps of: writing data from said computer processingoutput into a FIFO pipe; emptying said FIFO pipe into an intelligentprocess; using said intelligent process to distribute said data across asequence of data stores selected from a plurality of available datastores; emptying each of said data stores into said data storage deviceaccording to said sequence; and signaling said intelligent process whensaid sequence is completed.
 10. The method of claim 9 in which said stepof using said intelligent process further comprises the steps of:reading a list of available data stores; reading data output from saidFIFO pipe; detecting the end of said data; if said data has ended, thensignaling a data manager to empty a final data store; waiting for saiddata manager to signal completion; and recording a response code forsaid FIFO pipe; if said data has not ended, then selecting a currentdata store that is not busy; testing whether the current data store isfull; if full, then signaling the data manager to empty the current datastore; and marking the current data store as busy; repeating saidselecting and testing a current data store until a data store is foundto be not full and not busy and then proceeding as if not full; if notfull, then writing said data into said current data store; and repeatingsaid steps of reading data, detecting the end, selecting, testing, andwriting, until said end of data is detected.
 11. A method of directlyreading data from a storage device into a computer process comprisingthe steps of: activating a FIFO pipe having an input from an intelligentprocess and an output to a destination file; reading said data into saidintelligent process from said storage device; and writing said data intosaid FIFO pipe; wherein said intelligent process includes at least oneFIFO data store and each of said at least one FIFO data stores conveyssaid data from said storage device to said FIFO pipe.
 12. The method ofclaim 11 in which said step of reading said data further comprises thesteps of: identifying one of said at least one FIFO data stores to befilled; filling said identified data store with said data; signalingsaid intelligent process to identify a next FIFO data store; andrepeating the steps of filling, signaling, and identifying until saiddata from said storage device is exhausted; and signaling saidintelligent process that said data is exhausted.
 13. The method of claim11 in which said step of writing said data into said FIFO pope furthercomprises the steps of: obtaining a sequence of said at least one FIFOdata stores containing data; emptying said data into said FIFO pipe; andrepeating said steps of obtaining and emptying until a signal isreceived.
 14. The method of claim 11 in which said at least one FIFOdata store further comprises a plurality of data stores managed as around robin list, whereby each one of said plurality of data stores iseither identified for filling, waiting to be emptied, or available, asdetermined by said intelligent process.
 15. A method for communicatingbetween a computer and a storage device comprising the steps of: pipingthe output of a first computer process into an intelligent process;storing the output of said intelligent process into a first file in saidstorage device; reading data from a second file in a storage device intoan intelligent process; and piping said data into the input of a secondcomputer process; wherein said intelligent process includes a set ofFIFO data stores for conveying said output to said first file and forconveying said data to said second computer process.
 16. The method ofclaim 15 in which said step of storing further comprises the steps of:selecting one of said set of FIFO data stores and filling it with saidoutput; signaling said storage device to empty said output; andrepeating said steps of selecting and signaling until said firstcomputer process is complete.
 17. The method of claim 15 in which saidstep of reading data further comprises the steps of: selecting one ofsaid set of FIFO data stores and filling it with said data; signalingsaid intelligent process to empty said data; and repeating said steps ofselecting and signaling until said data file is exhausted.
 18. Themethod of claim 15 in which said intelligent process further includes: alist of FIFO data stores and a status indicator for each said FIFO datastore on said list; a sequencer for selecting and filling a FIFO datastore according to said status indicator; and a sequencer for selectingand emptying a FIFO data store according to said status indictor. 19.The method of claim 15 in which said intelligent process furtherincludes: a means for associating said output with said first file and ameans for associating said data with said second computer process.